1. Field of Invention
The present invention relates generally to super-resolution of images.
2. Related Art
Many applications, for example, such as high-definition television, forensic imaging, surveillance imaging, satellite imaging, medical and scientific imaging, use increasing amounts of resolution with great benefit. The effort to achieve ever increasing resolution in images runs into issues of cost and complexity in required optics and electronics. Also, reducing the pixel size in a sensor in order to increase the pixel density of an image, increases the effect of shot noise due to the lower amounts of light available per pixel.
By way of background, a video signal is a continuous flow of image frames. Each frame captures a temporal instant of a particular scene. The image frames may also have spatial differences between them, either due to motion of the camera or motion in a scene that is captured. Each image, for example, such as a low resolution (LR) image, is a representation of a scene with generally a substantial amount of noise. The noise may be due to information lost in image capture, such as low resolution imaging and other forms of electronic, or optical, noise that contribute to the general reduction in correspondence between the image and the actual scene.
Resolution enhancing techniques that use spatial interpolation—for example, bi-linear filtering, bi-cubic filtering, and poly-phase filtering—derive additional pixels for the high resolution (HR) image frame using the pixels of one low resolution image frame. The use of pixels in a single image frame to derive additional pixels for the high resolution image, generally results in a blurred image. For example, in bi-linear filtering, two adjacent pixels may be averaged to yield the value of a new pixel to be inserted between them: the new pixel being an average value of the two original pixels is likely to introduce some blurriness to the image.
Super-resolution of video signals is a technique by which an input video frame sequence at a low spatial resolution is transformed to an output video frame sequence of high spatial resolution through image processing. In contrast to spatial interpolation techniques, information from multiple low resolution frames are used to develop the high resolution frame. The use of multiple low resolution frame images gives super-resolution he ability to produce high resolution images with details not available in a single low resolution image. These high resolution images have more spatial detail, sharper edges, fewer artifacts such as blurring and aliasing, and less noticeable noise.
Super-resolution can be formulated as the inversion problem shown mathematically in equation (1). Equation (1) represents that an observed sequence of low resolution images of a scene, is derived from a high resolution image of the scene being effected by imaging process noise and additive random noise.Y=HX+N  (1),
where X represents the unknown high resolution image, Y represents the observed low resolution image, H is the system matrix of the imaging process, and N is the random additive noise. H represents the natural loss of spatial resolution caused, for example, due to optical distortions, motion blur, noise within the optics or electronics, noise introduced in transmission of the image, and insufficient sensor density. In super-resolution, generally, the objective is to find an estimate of the corresponding high resolution image X, from a set of observed images Y.
Several techniques are described in the art for super-resolution. Good overviews are provided in, Park, S. C., Park, M. K., and Kang, M. G., “Super-resolution Image Reconstruction: a technical overview,” IEEE Signal Processing Magazine, 20(3):21-36, May 2003; and Farsiu, S., Robinson, D., Elad, M., and Magazine, P., “Advances and Challenges in Super-Resolution,” International Journal of Imaging Systems and Technology, vol. 14, no. 2, pp. 47-57, August 2004. In general, super-resolution techniques can be categorized as either motion-based or motion-free. In motion-based techniques the attempt is to track an object in multiple low resolution images and then combine these spatially shifted versions of the object into a high resolution image of the object. In motion-free methods, one uses cues such as known corresponding samples of low resolution and high resolution images and edges to obtain high resolution details.
Motion-free techniques, such as the frequency-domain methods described in Tsai, R., and Huang, T., “Multiframe Image Restoration and Registration,” Advances in Computer Vision and Image Processing, vol. 5, issue 3, pp. 223-226, March 1987, rely oil global motion between low resolution frames. Other motion-free techniques, such as the learning-based methods described in Kepel, D., and Zisserman, A., “Super-resolution from multiple views using learnt image models,” Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 627-634, December 2001, require the development of an extensive database of mappings between low resolution and corresponding high resolution images.
Motion-based techniques that are described in the art include non-uniform sampling methods described in Keren, D., Peleg, S., and Brada, R., “Image Sequence Enhancement Using Subpixel Displacements,” Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 742-746, June 1998; projection onto convex sets (POCS) methods described in Stark, H., and Oskoui, P., “High-resolution image recovery from image-plane arrays using convex projections,” Journal of the Optical Society of America, A: Optics and Image Science, vol. 6, pp. 1715-1726, November 1989; bayesian methods described in Schultz, R., and Stevenson, R., “A Bayesian Approach to Image Expansion for Improved Definition,” IEEE Transactions on Image Processing, vol. 3, no. 3, pp. 233-242, May 1994; and, iterative back projection (IBP) methods or simulate-and-correct methods described in Peleg, S., Keren, D., and Schweitzer, D., “Improving Image Resolution by Using subpixel Motion,” Pattern Recognition Letters, vol. 5, issue 3, pp. 223-226, March 1987. Each one of these methods require a high level of computational complexity. In addition, POCS may require a-priori knowledge of some of the characteristics of the high resolution image, and bayesian methods may require a probability model that accurately describes the high resolution image.
What is needed, therefore, is a method of super-resolution imaging that is of reduced computational complexity that does not require a-priori knowledge of the desired high resolution image.